Predicting COVID-19 related death using the OpenSAFELY platform (preprint)

Objectives To compare approaches for obtaining relative and absolute estimates of risk of 28-day COVID-19 mortality for adults in the general population of England in the context of changing levels of circulating infection. Design Three designs were compared. (A) case-cohort which does not explicitly account for the time-changing prevalence of COVID-19 infection, (B) 28-day landmarking, a series of sequential overlapping sub-studies incorporating time-updating proxy measures of the prevalence of infection, and (C) daily landmarking. Regression models were fitted to predict 28-day COVID-19 mortality. Setting Working on behalf of NHS England, we used clinical data from adult patients from all regions of England held in the TPP SystmOne electronic health record system, linked to Office for National Statistics (ONS) mortality data, using the OpenSAFELY platform. Participants Eligible participants were adults aged 18 or over, registered at a general practice using TPP software on 1st March 2020 with recorded sex, postcode and ethnicity. 11,972,947 individuals were included, and 7,999 participants experienced a COVID-19 related death. The study period lasted 100 days, ending 8th June 2020. Predictors A range of demographic characteristics and comorbidities were used as potential predictors. Local infection prevalence was estimated with three proxies: modelled based on local prevalence and other key factors; rate of A&E COVID-19 related attendances; and rate of suspected COVID-19 cases in primary care. Main outcome measures COVID-19 related death. Results All models discriminated well between patients who did and did not experience COVID-19 related death, with C-statistics ranging from 0.92-0.94. Accurate estimates of absolute risk required data on local infection prevalence, with modelled estimates providing the best performance. Conclusions Reliable estimates of absolute risk need to incorporate changing local prevalence of infection. Simple models can provide very good discrimination and may simplify implementation of risk prediction tools in practice.

Predictive accuracy of computer-aided versions of the on-admission National Early Warning Score in estimating the risk of COVID-19 for unplanned admission to hospital: a retrospective development and validation study (preprint)

Objectives: To consider the potential of the National Early Warning Score (NEWS2) for COVID-19 risk prediction on unplanned admission to hospital. Design: Logistic regression model development and validation study using a cohort of unplanned emergency medical admission to hospital. Setting: York Hospital (YH) as model development dataset and Scarborough Hospital (SH) as model validation dataset. Participants: Unplanned adult medical admissions discharged over 3 months (11 March 2020 to 13 June 2020 ) from two hospitals (YH for model development; SH for external model validation) based on admission NEWS2 electronically recorded within 24 hours of admission. We used logistic regression modelling to predict the risk of COVID-19 using NEWS2 (Model M0') versus enhanced cNEWS models which included age + sex (model M1') + subcomponents (including diastolic blood pressure + oxygen flow rate + oxygen scale) of NEWS2 (model M2'). The ICD-10 code U071 was used to identify COVID-19 admissions. Model performance was evaluated according to discrimination (c statistic), calibration (graphically), and clinical usefulness at NEWS2 [≥]5. Results The prevalence of COVID-19 was higher in SH (11.0%=277/2520) than YH (8.7%=343/3924) with higher index NEWS2 (3.2 vs 2.8) but similar in-hospital mortality (8.4% vs 8.2%). The c-statistics for predicting COVID-19 for cNEWS models (M1',M2') was substantially better than NEWS2 alone (M0') in development (M2': 0.78 (95%CI 0.75-0.80) vs M0' 0.71 (95%CI 0.68-0.74)) and validation datasets (M2': 0.72 (95%CI 0.69-0.75) vs M0' 0.65 (95%CI 0.61-0.68)). Model M2' had better calibration than Model M0' with improved sensitivity (M2': 57% (95%CI 51%-63%) vs M0' 44% (95%CI 38%-50%)) and similar specificity (M2': 76% (95%CI 74%-78%) vs M0' 75% (95%CI 73%-77%)) for validation dataset at NEWS2[≥]5. Conclusions Model M2' is reasonably accurate for predicting the on-admission risk of COVID-19. It may be clinically useful for an early warning system at the time of admission especially to triage large numbers of unplanned hospital admissions.

Seek COVER: Development and validation of a personalized risk calculator for COVID-19 outcomes in an international network (preprint)

Abstract Importance COVID-19 is causing high mortality worldwide. Developing models to quantify the risk of poor outcomes in infected patients could help develop strategies to shield the most vulnerable during de-confinement. Objective To develop and externally validate COVID-19 Estimated Risk (COVER) scores that quantify a patient's risk of hospital admission (COVER-H), requiring intensive services (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis. Design Multinational, distributed network cohorts. Setting We analyzed a federated network of electronic medical records and administrative claims data from 13 data sources and 6 countries, mapped to a common data model. Participants Model development used a patient population consisting of >2 million patients with a general practice (GP), emergency room (ER), or outpatient (OP) visit with diagnosed influenza or flu-like symptoms any time prior to 2020. The model was validated on patients with a GP, ER, or OP visit in 2020 with a confirmed or suspected COVID-19 diagnosis across four databases from South Korea, Spain and the United States. Outcomes Age, sex, historical conditions, and drug use prior to index date were considered as candidate predictors. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. Results Overall, 43,061 COVID-19 patients were included for model validation, after initial model development and validation using 6,869,127 patients with influenza or flu-like symptoms. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, and kidney disease) which combined with age and sex could discriminate which patients would experience any of our three outcomes. The models achieved high performance in influenza. When transported to COVID-19 cohorts, the AUC ranges were, COVER-H: 0.73-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.82-0.90. Calibration was overall acceptable, with overestimated risk in the most elderly and highest risk strata. Conclusions and relevance A 9-predictor model performs well for COVID-19 patients for predicting hospitalization, intensive services and death. The models could aid in providing reassurance for low risk patients and shield high risk patients from COVID-19 during de-confinement to reduce the virus' impact on morbidity and mortality.

Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection (preprint)

Objective To review and critically appraise published and preprint reports of models that aim to predict either (i) presence of existing COVID-19 infection, (ii) future complications in individuals already diagnosed with COVID-19, or (iii) models to identify individuals at high risk for COVID-19 in the general population. Design Rapid systematic review and critical appraisal of prediction models for diagnosis or prognosis of COVID-19 infection. Data sources PubMed, EMBASE via Ovid, Arxiv, medRxiv and bioRxiv until 24th March 2020. Study selection Studies that developed or validated a multivariable COVID-19 related prediction model. Two authors independently screened titles, abstracts and full text. Data extraction Data from included studies were extracted independently by at least two authors based on the CHARMS checklist, and risk of bias was assessed using PROBAST. Data were extracted on various domains including the participants, predictors, outcomes, data analysis, and prediction model performance. Results 2696 titles were screened. Of these, 27 studies describing 31 prediction models were included for data extraction and critical appraisal. We identified three models to predict hospital admission from pneumonia and other events (as a proxy for covid-19 pneumonia) in the general population; 18 diagnostic models to detect COVID-19 infection in symptomatic individuals (13 of which were machine learning utilising computed tomography (CT) results); and ten prognostic models for predicting mortality risk, progression to a severe state, or length of hospital stay. Only one of these studies used data on COVID-19 cases outside of China. Most reported predictors of presence of COVID-19 in suspected patients included age, body temperature, and signs and symptoms. Most reported predictors of severe prognosis in infected patients included age, sex, features derived from CT, C-reactive protein, lactic dehydrogenase, and lymphocyte count. Estimated C-index estimates for the prediction models ranged from 0.73 to 0.81 in those for the general population (reported for all 3 general population models), from 0.81 to > 0.99 in those for diagnosis (reported for 13 of the 18 diagnostic models), and from 0.85 to 0.98 in those for prognosis (reported for 6 of the 10 prognostic models). All studies were rated at high risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, and poor statistical analysis, including high risk of model overfitting. Reporting quality varied substantially between studies. A description of the study population and intended use of the models was absent in almost all reports, and calibration of predictions was rarely assessed. Conclusion COVID-19 related prediction models are quickly entering the academic literature, to support medical decision making at a time where this is urgently needed. Our review indicates proposed models are poorly reported and at high risk of bias. Thus, their reported performance is likely optimistic and using them to support medical decision making is not advised. We call for immediate sharing of the individual participant data from COVID-19 studies to support collaborative efforts in building more rigorously developed prediction models and validating (evaluating) existing models. The aforementioned predictors identified in multiple included studies could be considered as candidate predictors for new models. We also stress the need to follow methodological guidance when developing and validating prediction models, as unreliable predictions may cause more harm than benefit when used to guide clinical decisions. Finally, studies should adhere to the TRIPOD statement to facilitate validating, appraising, advocating and clinically using the reported models. Systematic review registration protocol: osf.io/ehc47/, registration: osf.io/wy245